Method and system for efficiently storing web pages for quick downloading at a remote device

ABSTRACT

A web server for efficiently storing web pages for quick downloading at a remote device comprises a fast memory, a disk memory and a storage device. The fast memory stores a first plurality of web pages and receives a first signal from the remote device indicating selection of one of the first plurality of web pages. The disk memory stores a second plurality of web pages linked to the first plurality of web pages and transfers all of the second plurality of web pages that are linked to the selected one of the first plurality of web pages to the fast memory in response to the first signal. The disk memory also transmits a second signal to the storage device in response to the first signal. The storage device stores a third plurality of web pages linked to the first plurality of web pages and transfers all of the third plurality of web pages that are linked to the selected ones of the second plurality of web pages to the disk memory in response to the second signal. In addition, the fast memory deletes each of the non-selected first plurality of web pages in response to the first signal so as to allow for memory reuse and to aid in quickly downloading selected web pages to the remote device.

TECHNICAL FIELD

This invention relates to methods and systems for efficiently storingweb pages at a server for quick downloading at a remote device incommunication with the server.

BACKGROUND ART

Traditional web servers depend on sequential access to HTML (HyperTextMarkup Language) pages by loading hyperlinked pages as the clientrequests them while navigating through the current page. Each clientinitiated hit involves a random access to the memory, even if multipleclients access the same page. Thus, overall latency is high even whenthere is an overlap in the access. Latency refers to the amount of timeit takes a web server to respond to a client beginning from the time therequest was sent by the client to the server. To reduce latency in thenetwork, some proposals have been made to have local machines, mostlyproxy servers, to prefetch the pages linked to the current page beforethe client actually “hits” them. However, since all web pages may not berequested by the client, storage is wasted. Furthermore, bandwidth isalso wasted. If a user is on a public area network, the user will payfor the time of access or number of bytes transferred every time he/sheuses the bandwidth. Consequently, if the network is busy, aggressiveprefetching can be expensive.

File servers, on the other hand, typically utilize a hierarchicalstorage manager (HSM) in managing access of data stored on disk andtape. An application program in an HSM configuration examines memoryusage in the hard drive. Instead of putting all files, or data, on thehard drive, the program will keep a subset of the files on the harddrive and the rest on the tape. By ensuring that the most currently orfrequently used files are on the hard drive, most accesses are satisfiedby accesses to the hard drive, i.e., hits.

If the file required is not on the hard drive, i.e., a miss, a memoryrequest is sent to the tape. Since the tape is slower than the disk, thetime to retrieve the file is higher. Since the access is usually to thedisk, this penalty is only apparent on a small percentage of theaccesses. Thus, the main motivation for using HSM is cost. Tape issignificantly cheaper and denser than a hard drive. However, in somefile requests, latency will suffer.

Thus, there exists a need for a method and system for efficientlystoring scaleable amounts of data at a web server that can be quicklyaccessed by a client without compromising bandwidth.

DISCLOSURE OF THE INVENTION

It is a general object of the present invention to provide a method andsystem for efficiently storing web pages at a server for quickdownloading at a remote device in communication with the server.

In carrying out the above object and other objects, features, andadvantages of the present invention, a method is provided forefficiently storing web pages for quick downloading at a remote devicein a computer network including a server computer having a fast memoryfor storing a plurality of home web pages, a disk memory for storing afirst plurality of web pages linked to the home web pages, and a storagedevice coupled to the computer fore storing a second plurality of webpages linked to the home web pages. The method includes the step ofreceiving a first signal from the remote device at the server computerindicating selection of one of the plurality of home pages. The methodalso includes the step of transferring at least one of the firstplurality of web pages linked to the selected home page from the diskmemory to the fast memory in response to the first signal. Stillfurther, the method includes the step of transmitting a second signalfrom the disk memory to the storage device in response to the firstsignal. Finally, the method includes the step of transferring at leastone of the second plurality of web pages linked to the selected homepage from the storage device to the disk memory in response to thesecond signal wherein anticipated web pages linked to the selected homepage are quickly accessible by the remote device.

In further carrying out the above object and other objects, features,and advantages of the present invention, a web server is also providedfor carrying out the steps of the above described method. The web serverincludes a fast memory for storing a plurality of home web pages and forreceiving a first signal from the remote device indicating selection ofone of the plurality of home pages. The web server also includes a diskmemory for storing a first plurality of web pages linked to the home webpages and for transferring at least one of the first plurality of webpages linked to the selected home page to the fast memory in response tothe first signal. The disk memory is further provided for transmitting asecond signal in response to the first signal. The web server furtherincludes a storage device for storing a second plurality of web pageslinked to the home web pages, receiving the second signal, andtransferring at least one of the second plurality of web pages linked tothe selected home page to the disk memory in response to the secondsignal wherein anticipated web pages linked to the selected home pageare quickly accessible by the remote device.

The above object and other objects, features and advantages of thepresent invention are readily apparent from the following detaileddescription of the best mode for carrying out the invention when takenin connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of the storage-efficient web server of thepresent invention;

FIG. 2 is a directed page graph for a page sequence that begins withpage i;

FIG. 3 is an example of a page distribution in the server storagehierarchy; and

FIG. 4 is a flow diagram illustrating the general sequence of stepsassociated with the present invention.

BEST MODES FOR CARRYING OUT THE INVENTION

A schematic diagram of the storage-efficient web server is shown in FIG.1, denoted generally by reference numeral 10. The server 10 includes acomputer 12 such as, for example, a workstation or a high-end personalcomputer. The server 10 hosts a plurality of web sites, wherein each website includes a plurality of HTML documents, or web pages.

The computer 12 of the server 10 includes a fast memory 14, such ascache or RAM (random access memory), and a disk memory 16. Althoughshown externally, the disk memory 16 may be either an internal diskdrive or an externally attached disk or disk farm, such as a RAID(redundant array of inexpensive disks). Both the fast memory 14 and thedisk memory 16 store a portion of the web pages for quick downloading aswill be described in greater detail below.

The computer 12 is also connected to an external storage device 18, suchas tape or removable media, which stores even more data relating to theweb pages. The storage device 18 can be connected to the computer 12either directly via a point-to-point link, as shown in FIG. 1, orindirectly via a storage area network. The server 10 is in communicationwith a plurality of remote devices 20 which can download any of the webpages stored at the server 10. Examples of such devices include acomputer, an Internet-ready telephone, etc.

As shown in FIG. 1, a three-tier hierarchy exists; fast memory 14, diskmemory 16 and storage device 18, where the traditional assumption oflower access speed, larger capacity, and lower cost holds true as onemoves down the hierarchy. While in most cases these may represent cacheor RAM, disk and tape, any other forms of storage that satisfies theaccess speed, capacity and cost monotonicity also applies. In any case,the storage hierarchy, which can be extended to more than three tiers,forms a serial buffer chain where the cache in fast memory 14 issupported by a cache in disk memory 16 which in turn is supported by thedata in the storage device 18.

Also, this chained buffer configuration allows the possibility of datato be transferred between fast memory 14 and disk memory 16 whilestorage device 18 transfers data to disk memory 16, provided disk memory16 utilizes a typical double buffer so that one buffer feeds disk memory16. while the second buffer is filled by storage device 18. Thissimultaneous data transfer between hierarchies is possible as long asdisk memory 16 possesses adequate data bandwidth and connectivity eitherthrough direct point-to-point links or through a storage area network.

The server 10 of the present invention reduces cache space at fastmemory 14 storage level which results in fewer pages of a web site beingput on the computer 12 while off-loading successive pages in the secondlevel storage of the storage device 18. Before continuing with thedescription of the present invention, an exemplary directed page graphis shown in FIG. 2 for a page sequence that begins with page i. Notethat because the page numbering convention is visit order independent,the naming sequence is not unique. Therefore, many different page graphscan result in the same page number sequence.

Returning to the discussion of the present invention, in order to reducethe cache space at the fast memory 14, a large number of the first levelhome pages, i.e., the first page of the web sites, i, j, k, etc. arestored in the fast memory 14, as shown in FIG. 3. Disk memory 16contains a larger sequence of subsequent levels of the web site pages,i.e., i, i₁₁ and i₁₂, where i_(jk) refers to the kth hyperlinked page onthe jth level.

First level home pages, e.g., i, are kept in fast memory 14 while thepages i₁₁ and i₁₂, hyperlinked by i, are kept in disk memory 16. Thecache in fast memory 14 has links to locations on disk memory 16, asshown by the connections in FIG. 3. That is, when cached page i isaccessed from fast memory 14, a pointer points to the location (address)of the same page i in disk memory 16. Similarly, the sequence for pagesi, i₁₁, i₁₂ in disk memory 16 reference the address in storage device 18so that on further access (beyond i₁₂ in disk memory 16), the server 10knows where to fetch the rest of the pages from storage device 18.

When there is a hit on i in fast memory 14, all pages linked to i indisk memory 16 are read into fast memory 14 even before there is a hiton any of i₁₁ or i₁₂. Therefore, while the page is sent to the client,fast memory 14 reads in i₁₁ and i₁₂, according to the example shown inFIG. 2. Thus, prefetching of certain pages from disk memory 16 reducesthe total number of home pages that normally resides in fast memory 14.By using cache space reuse, described below, the space in fast memory 14can be kept relatively small for each web site hosted.

Cache space reuse implies that the cache used in fast memory 14 is keptto a fixed maximum per home page. This is achieved by prefetching fromdisk memory 16, discarding pages not required in the cache for fastmemory 14 and reading in pages from storage device 18.

To follow on the simple example described above with reference to FIGS.2 and 3, the following sequence is executed, as shown in FIG. 4. In thefollowing example, it is assumed that disk memory 16 stores only thefirst level of web pages linked to the home page, while storage device18 stores the remaining levels of web pages linked to the home page. Thestorage space of disk memory 16 actually depends on the access time oflocating partitions in storage device 18, i.e., finding the address instorage device 18 where the pages are written. For example, an addressutilized in a disk-type storage device 18 is identified by a {sector,track, block number} descriptor. If storage device 18 is very fast, theneither only the first level or the first and second level pages onlycould be stored in disk memory 16. A further explanation of the storagespace of disk memory 16 is described in greater detail below.

In continuing with the example, a first signal is transmitted from aclient at one of the remote devices 20 requesting one of the home pages,e.g., i, as shown at block 100. The server computer 12 receives thefirst signal, as shown at block 110.

In response to the first-signal, disk memory 16 transfers the firstlevel web pages linked to the selected home page, i.e., i₁₁ and i₁₂, tofast memory 14, as shown at block 112. At the same time, disk memory 16transmits a second signal to storage device 18 requesting specific data,as shown at block 114. In response to the second signal, storage device18 transfers the second level web pages linked to the selected-home pageto disk memory 16, i.e., i₂₁, i₂₂ and i₂₃, as shown at block 116.

If the client requests one of the first level web pages, e.g., i₁₂, athird signal is transmitted from the remote device 20 accordingly, asshown at block 118. Fast memory 14 then deletes the remainingnon-selected first level web pages, i.e., page i₁₁, as shown at block120, and sends the selected web page to the client. Since disk memory 16has read in and stored pages i₂₁, i₂₂, i₂₃, etc., from storage device18, disk memory 16 transfers the second level web pages related to pagei₁₂, i.e., page i₂₃, to fast memory 14. Similarly, disk memory 16 maydelete the non-selected second level web pages if cache memory size is aconcern. Then, disk memory 16 transmits a fourth signal to storagedevice 18 requesting the web pages linked to the selected page i₂₃,i.e., pages i₃₃ and i₃₄. The method continues in a like manner as theclient requests additional web pages.

It is possible that the client may proceed to traverse the web pages ina reverse order back to a previous web page where there were more thanone possible path from that web page, e.g., page i utilizing the exampleshown in FIG. 2. In this instance, upon reaching that web page, diskmemory 16 transmits a signal to storage device 18 requesting the webpages linked to that page, e.g. pages i₁₁ and i₁₂, as described at block112. Thus, disk memory 16 anticipates the client's request by insuringfast memory 14 always has all of the first level web pages-related tothe selected web page stored therein. At the same time, disk memory 16insures that all of the second level web pages related to the selectedweb page are stored in disk memory. This is achieved via memory re-useas described above.

The above sequence depends on the access time of locating partitions instorage device 18, wherein partitions correspond to the portion of thestorage device 18 that comprises all the web pages and sequences of theassociated web site. If the time to access a partition in storage device18 is t_(M2), then the number of pages to be kept in M1 will bedependent on two factors: i) average size, s(i_(jk)), of an individualpage i_(jk) at any level j in the page directed graph, and ii) reactiontime of the client, assumed fixed t_(r). The level j of the number ofpages to be hosted in disk memory 16 is determined according to thefollowing:$j \ni {{{\sum\limits_{j}\quad t_{p}} + t_{r} + \frac{\overset{\_}{s\left( i_{jk} \right)}}{L}} \geq t_{M2}}$where t_(p) is the average propagation delay between client and server,L is the average link bandwidth, or effective data rate, between theclient and server 10, and {overscore (s(i)}_(jk)) over k represents theaverage size of the pages at level j.

A simple explanation of the limit to j is as follows. Each level of thetree traversed (i.e., j=1, 2, etc.) provides the server 10 the followingtime window consisting of the client reaction time, t_(r), which isprobably 0.25 seconds for the average human, the delay or propagationtime, t_(p), and the time of flight {overscore (s(i)}_(jk))/L to sendthe average page at level j to the client to access storage device 18.As the average page size {overscore (s(i)}_(jk)) increases, its time offlight provides a larger time window during which access to a slowstorage device 18 can be masked due to the limited link bandwidth L. Thepropagation delay is typically ignored since the client can be at anyarbitrary distance from the server 10.

The above inequality is somewhat conservative since it assumes that theaverage page size is a good indicator of the time of flight. In the casewhere there is a large variation in file size, the mean should bereplaced with a minimum, Min(s(i_(jk))). This results in increasing thedepth of the pages to be stored and potentially increases the storagerequired in disk memory 16.

An example illustrating the use of the above limit is as follows. If thelink bandwidth L to the server 10 is 56 Kbs, the average page size ateach level in the page directed graph of the home page sequence is 50KB, and the access time to storage device 18 is eight seconds, then atleast two levels of pages (j=2) must be resident in disk memory 16. Incase of the i page sequence, pages i, i₁₁, i₁₂, i₂₁, i₂₂ and i₂₃ must bekept in disk memory 16.

Typically, however, the average page loaded on the Web is only 6 KB.This implies that a larger depth (level) of pages may be stored in diskmemory 16 for a fixed size cache. However, since the pages are ofsmaller size, the total storage required (i.e., number of pages perlevel*average size per page*number of levels) will not be significantlydifferent.

To ensure that successive pages can be read from storage device 18 withno perceived latency, a quasi-sequential access of the levels isutilized, i.e., page i_(j+l,l) is always read after page i_(j,k) evenwhen k<1. If i_(j,k)=i11, then i_(j+l,l=i) ₂₁, where l is any integer 1,2, 3, . . . , etc. Access to web site pages is not totally random sinceonly certain paths can be traversed by a client, nor is the accesstotally sequential since each web site typically has multiple yet afinite number of links on each level. By exploiting thisquasi-sequentiality, slow sequential access storage devices can beexploited.

All home page sequences are added at a boundary of the partition so thatthe partition can be accessed with moderately low deterministic accesstime. Vertical partitioning works well in this case, i.e. data storedsequentially. If access time increases to further out partitions, onecan place home pages with very large pages at the further ends and homepages with small pages at the beginning of the device 18, i.e., verticalpartitioning. In vertical partitioning, each partition starts from thesame starting point and proceeds from left to right so that the accesstime to the first page on any partition is zero whenever the tape isread.

Clearly, storage devices 18 that exhibit low partition access latencywill be well-suited to developing hierarchical storage web servers asdisclosed herein with very large capacities and with the added benefitof fully masking latencies to disk memory 16 and storage device 18.

However, the storage device 18 may be a non-sequential device, such as,for example, a random access optical disk. In this case, the presentinvention is still valid even though partitioning as described above isnot necessary.

While the best modes for carrying out the invention have been describedin detail, those familiar with the art to which this invention relateswill recognize various alternative designs and embodiments forpracticing the invention as defined by the following claims.

1-12. (canceled)
 13. In a computer network including a server computerhaving a first storage device for storing a first plurality of web pagesand a second storage device for storing a second plurality of web pageslinked to the first plurality of web pages, a method for efficientlystoring web pages for quick downloading at a remote device, the methodcomprising: receiving a first signal from the remote device at theserver computer indicating selection of one of the first plurality ofweb pages; and transferring all of the second plurality of web pagesthat are linked to the selected one of the first plurality of web pagesfrom the second storage device to the first storage device in responseto the first signal so that anticipated web pages linked to any webpages selected by the remote device are quickly accessible by the remotedevice.
 14. The method of claim 13 wherein the first storage devicecomprises a fast memory and the second storage device comprises a diskmemory.
 15. The method of claim 13 wherein transferring all of thesecond plurality of web pages includes deleting each of the non-selectedfirst plurality of web pages in the first storage device.
 16. The methodof claim 13 wherein storing the second plurality of web pages linked tothe first plurality of web pages includes: determining one of an averagesize and a minimum size of each web page linked to each of the firstplurality of web pages; and determining a reaction time of a clientoperating the remote device.
 17. The method of claim 16 furthercomprising: determining an average propagation delay between the clientand the server; determining an average link bandwidth between the clientand the web server; and determining a time to access a partition in thestorage device.
 18. A system for providing quick downloading of webpages at a remote device, the system comprising: a server computer; afirst storage device in communication with the server computer forstoring a first plurality of web pages and for receiving a first signalfrom the remote device indicating selection of one of the firstplurality of web pages; and a second storage device in communicationwith the first storage device, the second storage device for storing asecond plurality of web pages linked to the first plurality of web pagesand for transferring all of the second plurality of web pages that arelinked to the selected one of the first plurality of web pages to thefirst storage device in response to the first signal so that anticipatedweb pages linked to any web pages selected by the remote device arequickly accessible by the remote device.
 19. The system of claim 18wherein the first storage device comprises a fast memory and the secondstorage device comprises a disk memory.
 20. The system of claim 18wherein the first storage device is further operative to delete each ofthe non-selected first plurality of web pages in response to the firstsignal.
 21. The system of claim 18 wherein the second storage device, instoring the second plurality of web pages linked to the first pluralityof web pages, stores the second plurality of web pages based on one ofan average size and a minimum size of each web page linked to each ofthe first plurality of web pages and a reaction time of a clientoperating the remote device.
 22. The system of claim 21 wherein thesecond storage device is further operative to store the second pluralityof web pages based on an average propagation delay between the clientand the server, an average link bandwidth between the client and theserver, and an access time to access a partition in the storage device.23. The system of claim 18 wherein the first storage device is a cache.24. The system of claim 18 wherein the first storage device is a randomaccess memory.
 25. The system of claim 18 wherein the second storagedevice is an internal memory.
 26. The system of claim 18 wherein thesecond storage device is an external memory.
 27. The system of claim 18wherein the second storage device is a hard drive.
 28. The system ofclaim 18 wherein the second storage device is a redundant array ofinexpensive disks.
 29. In a computer network including a server computerhaving a first storage device for storing a first plurality of web pagesand a second storage device for storing a second plurality of web pageslinked to the first plurality of web pages, the first and second storagedevices arranged in a hierarchy wherein the second storage device has alower access speed and a larger capacity than the first storage device,a method for efficiently storing web pages for quick downloading at aremote device, the method comprising: receiving a first signal from theremote device at the server computer indicating selection of one of thefirst plurality of web pages; and transferring all of the secondplurality of web pages that are linked to the selected one of the firstplurality of web pages from the second storage device to the firststorage device in response to the first signal so that anticipated webpages linked to any web pages selected by the remote device are quicklyaccessible by the remote device.
 30. The method of claim 29 wherein thefirst storage device comprises a fast memory and the second storagedevice comprises a disk memory.
 31. The method of claim 29 whereintransferring all of the second plurality of web pages includes deletingeach of the non-selected first plurality of web pages in the firststorage device.
 32. The method of claim 29 wherein storing the secondplurality of web pages linked to the first plurality of web pagesincludes: determining one of an average size and a minimum size of eachweb page linked to each of the first plurality of web pages; anddetermining a reaction time of a client operating the remote device. 33.The method of claim 32 further comprising: determining an averagepropagation delay between the client and the server; determining anaverage link bandwidth between the client and the web server; anddetermining a time to access a partition in the storage device.
 34. Asystem for providing quick downloading of web pages at a remote device,the system comprising: a server computer; a first storage device incommunication with the server computer for storing a first plurality ofweb pages and for receiving a first signal from the remote deviceindicating selection of one of the first plurality of web pages; and asecond storage device in communication with the first storage device,the second storage device having a lower access speed and a largercapacity than the first storage device, the second storage device forstoring a second plurality of web pages linked to the first plurality ofweb pages and for transferring all of the second plurality of web pagesthat are linked to the selected one of the first plurality of web pagesto the first storage device in response to the first signal so thatanticipated web pages linked to any web pages selected by the remotedevice are quickly accessible by the remote device.
 35. The system ofclaim 34 wherein the first storage device comprises a fast memory andthe second storage device comprises a disk memory.
 36. The system ofclaim 34 wherein the first storage device is further operative to deleteeach of the non-selected first plurality of web pages in response to thefirst signal.
 37. The system of claim 34 wherein the second storagedevice, in storing the second plurality of web pages linked to the firstplurality of web pages, stores the second plurality of web pages basedon one of an average size and a minimum size of each web page linked toeach of the first plurality of web pages and a reaction time of a clientoperating the remote device.
 38. The system of claim 37 wherein thesecond storage device is further operative to store the second pluralityof web pages based on an average propagation delay between the clientand the server, an average link bandwidth between the client and theserver, and an access time to access a partition in the storage device.39. The system of claim 34 wherein the first storage device is a cache.40. The system of claim 34 wherein the first storage device is a randomaccess memory.
 41. The system of claim 34 wherein the second storagedevice is an internal memory.
 42. The system of claim 34 wherein thesecond storage device is an external memory.
 43. The system of claim 34wherein the second storage device is a hard drive.
 44. The system ofclaim 34 wherein the second storage device is a redundant array ofinexpensive disks.